05. Demo: Defining DQN Architecture

Part 1

Cd13650 C5 L3 Demo 1 V2

Building a Trading Agent with DQN

Learn the foundational steps to construct a trading agent using Deep Q-Networks (DQN). This guide breaks down the initial components and processes using Python libraries and TensorFlow with Keras.

Key Steps and Concepts:

  1. Set-Up and Initialization

    • Import essential Python libraries.
    • Define functions for data display and reproducibility.

  2. Prepare Sample Data

    • Load and clean the sample data.
    • Ensure necessary features are included but not normalized for simplicity.

  3. Define the DQN Model Architecture

    • Use TensorFlow with Keras to create a sequential model.
    • Create three layers: Input, Hidden, and Output.
      • Use ReLU activation for the Input and Hidden layers.
      • Implement a linear activation for the Output layer.
    • Compile the model using mean squared error loss and the Adam optimizer.

  4. Agent Class Implementation

    • Define critical parameters such as Window Size, Feature Count, and Action Size.
    • Set up Memory and Inventory to manage data for training and trading.
    • Introduce parameters like Gamma and Epsilon, crucial for experience replay.
    • Enable testing mode for loading pre-trained models efficiently.

Outcome:

Build a functional trading agent ready to be trained or tested using historical data, ensuring adaptability through parameter customization and model saving options.

Part 2

Cd13650 C5 L3 Demo 1b V2

Integrating DQN into the Agent Class

This segment showcases the integration and configuration of a Deep Q-Network (DQN) within an agent class for reinforcement learning. The focus is on defining models within training and testing modes and creating helper functions.

Training and Testing Mode Setup

  • Model Initialization: DQN is initialized using state size and action size from the agent class.
  • Model Access: Direct access is enabled to the model through self.model for seamless operations.

Helper Functions

  • Get Q-Values for State:
    • Simplifies the prediction process by renaming the built-in predict function.
    • Handles input state reshaping to easily retrieve q_values.

Model Fitting Process

  • Fitting Functionality:
    • Employs gradient descent using mean squared error.
    • Uses the Adam optimizer with a 0.001 learning rate.
    • Adjusts model based on input state and target Q-table.

These steps aim to simplify the implementation and ensure seamless application of reinforcement learning strategies within the training setup. Future topics will include completing the agent class with act and experience replay functions.

Next Concept